System and method of highlighting influential samples in sequential analysis

ABSTRACT

Attention weights in a hierarchical attention network indicate the relative importance of portions of a conversation between an individual at one terminal and a computer or a human agent at another terminal. Weighting the portions of the conversation after converting the conversation to a standard text format allows for a computer to graphically highlight, by color, font, or other indicator visible on a graphical user interface, which portions of a conversation led to an escalation of the interaction from an intelligent virtual assistant to a human customer service agent.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 16/283,135, now U.S. Pat. No. 11,048,854, whichclaims priority to and incorporates entirely by reference correspondingU.S. Provisional Patent Application Ser. No. 62/633,827 filed on Feb.22, 2018.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

None.

FIELD

The objective of this disclosure is to create visual alerts for humancustomer service representatives receiving calls from customers thathave been forwarded from intelligent virtual assistants (IVA's).

BACKGROUND

In (Yang et al., 2016), a hierarchical attention network (HAN) waspreviously created for document classification. The attention layer isused to visualize words influential in classifying the document toexplain the model's prediction. The information below regarding thebackground on HANs is quoted largely from Zichao Yang, Diyi Yang, ChrisDyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy, 2016,Hierarchical Attention Networks for Document Classification, inHLT-NAACL. pages 1480-1489, which is incorporated by reference as if setforth fully herein.

Hierarchical Attention Networks

The overall architecture of the Hierarchical Attention Network (HAN) isshown in Prior Art FIG. 2. It consists of several parts: a word sequenceencoder, a word-level attention layer, a sentence encoder and asentence-level attention layer. This background describes the details ofdifferent components in the following sections.

GRU-Based Sequence Encoder

The gated recurrent unit (“GRU”) explained by Bandanau et al. isincorporated by reference as if set forth fully herein and can be foundat Dzmitry Bandanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neuralmachine translation by jointly learning to align and translate. arXivpreprint arXiv: 1409-0473. It uses a gating mechanism to track the stateof sequences without using separate memory cells. There are two types ofgates: the reset gate rt and the update gate zt. They together controlhow information is updated to the state. At time t, the GRU computes thenew state as

h _(t)=(1−z _(t))⊙h _(t-1) +z _(t) ⊙{tilde over (h)} _(t).  (1)

This is a linear interpolation between the previous state h(t−1) and thecurrent new state t computed with new sequence information. The gate ztdecides how much past information is kept and how much new informationis added. zt is updated as:

z _(t)=σ(W _(z) x _(t) +U _(z) h _(t-1) +b _(z)),  (2)

where xt is the sequence vector at time t. The candidate state t iscomputed in a way similar to a traditional recurrent neural network(RNN):

{tilde over (h)} _(t)=tanh(W _(h) x _(t) +r _(t)⊙(U _(h) U _(t-1))+b_(tx)),  (3)

Here rt is the reset gate which controls how much the past statecontributes to the candidate state. If rt is zero, then it forgets theprevious state. The reset gate is updated as follows:

Hierarchical Attention

This background regarding hierarchical attention networks focuses ondocument-level classification in this work. Assume that a document has Lsentences si and each sentence contains Ti words, and “w_(it)” witht∈[1, T] represents the words in the i^(th) sentence. The HAN modelprojects the raw document into a vector representation, on which one canbuild a classifier to perform document classification. In the following,this background of HAN analysis will present how to build the documentlevel vector progressively from word vectors by using the hierarchicalstructure.

Word Encoder

Given a sentence with words w_(it), t∈[0, T], the described embodimentsfirst embed the words to vectors through an embedding matrix We,x_(ij)=w_(e)w_(ij). This background explains a bidirectional GRU(Bandanau et al., 2014) to get annotations of words by summarizinginformation from both directions for words, and therefore incorporatethe contextual information in the annotation. The bidirectional GRUcontains the forward GRU f→which reads the sentence si from wi₁ towi_(T) and a backward GRU f←which reads from wi_(T) to wi₁:

x _(it) =W _(e) w _(it) ,t∈[1,T],

{right arrow over (h)} _(it) ={right arrow over (GRU)}(x _(it)),t∈[1,T],

_(it)=

(x _(it)),t∈[T,1].

Next, the model obtains an annotation for a given word with w_(it) byconcatenating the forward hidden state h→it and backward hidden stateh←it, i.e., h_(it)=[h→it, h←it], which summarizes the information of thewhole sentence centered around “w_(it).”

Note this directly uses word embeddings.

Word Attention

Not all words contribute equally to the representation of the sentencemeaning. Hence, this summary introduces an attention mechanism toextract such words that are important to the meaning of the sentence andaggregate the representation of those informative words to form asentence vector. Specifically,

u_(it) = tanh (W_(w)h_(it) + b_(w))$\alpha_{it} = \frac{\exp\left( {u_{it}^{T}u_{w}} \right)}{\Sigma_{t}{\exp\left( {u_{it}^{T}u_{w}} \right)}}$$s_{i} = {\sum\limits_{i}{\alpha_{it}{h_{it}.}}}$

That is, one first feeds the word annotation h_(it) through a one-layerMLP to get u_(it) as a hidden representation of h_(it), then thedescribed embodiments measure the importance of the word as thesimilarity of u_(it) with a word level context vector u_(w) and get anormalized importance weight αit through a “softmax” function of theprior art to normalize the vector into probability distributions. Afterthat, the described embodiments compute the sentence vector s_(i) as aweighted sum of the word annotations based on the weights. The contextvector u_(w) can be seen as a high-level representation of a fixed query“what is the informative word” over the words like that used in memorynetworks. The word context vector u_(w) is randomly initialized andjointly learned during the training process.

Sentence Encoder Given the sentence vectors s_(i), the describedembodiments can get a document vector in a similar way. the describedembodiments use a bidirectional GRU to encode the sentences:

{right arrow over (h)} _(i) ={right arrow over (GRU)}(s _(i)),i∈[1,L],

_(i)=

(s _(i)),t∈[L,1].

This background HAN concatenates h→i and h←j to get an annotation ofsentence i, i.e., hi=[h→i, h←i]. The variable “hi summarizes theneighbor sentences around sentence i but still focus on sentence i.

Sentence Attention

To reward sentences that are clues to correctly classify a document, thedescribed embodiments again use attention mechanism and introduce asentence level context vector us and use the vector to measure theimportance of the sentences. This yields where v is the document vectorthat summarizes all the information of sentences in a document.Similarly, the sentence level context vector can be randomly initializedand jointly learned during the training process.

$\begin{matrix}{{u_{i} = {\tanh\left( {{W_{s}h_{i}} + b_{s}} \right)}},} & (8) \\{{\alpha_{i} = \frac{\exp\left( {u_{i}^{T}u_{s}} \right)}{\Sigma_{i}{\exp\left( {u_{i}^{T}u_{s}} \right)}}},} & (9) \\{{v = {\sum\limits_{i}{\alpha_{i}h_{i}}}},} & (10)\end{matrix}$

where v is the document vector that summarizes all the information ofsentences in a document. Similarly, the sentence level context vectorcan be randomly initialized and jointly learned during the trainingprocess.

Document Classification

The document vector v is a high level representation of the document andcan be used as features for document classification:

p=softmax(W _(c) v+b _(c)).

One uses the negative log likelihood of the correct labels as trainingloss:

${L = {- {\sum\limits_{d}{\log\mspace{14mu} p_{dj}}}}},$

where j is the label of document d.

LSTM Background

Unlike the above described gated recurrent units (GRUs), long short termmemory cells (LSTMs) described below make small modifications to theinformation by multiplications and additions. This structure is wellexplained at Pranjal Srivastavahttps://www.analyticsvidhya.com/blog/2017/12/fundamentals-of-deep-learning-introduction-to-lstm,December 2017, which is incorporated by reference as if set forth fullyherein.

With LSTMs, the information flows through a mechanism known as cellstates. This way, LSTMs can selectively remember or forget things. Theinformation at a particular cell state has three dependencies.

These dependencies can be generalized to any problem as:

The previous cell state (i.e. the information that was present in thememory after the previous time step);

The previous hidden state (i.e. this is the same as the output of theprevious cell);

The input at the current time step (i.e. the new information that isbeing fed in at that moment).

A typical LSTM network is comprised of different memory blocks calledcells. There are two states that are being transferred to the next cell;the cell state and the hidden state. The memory blocks are responsiblefor remembering things and manipulations to this memory is done throughthree major mechanisms, called gates. Each of them is being discussedbelow.

Forget Gate

A forget gate is responsible for removing information from the cellstate. The information that is no longer required for the LSTM tounderstand things or the information that is of less importance isremoved via multiplication of a filter.

Input Gate

The input gate is responsible for the addition of information to thecell state.

Output Gate

This job of selecting useful information from the current cell state andshowing it out as an output is done via the output gate.

The above noted gates allow LSTMs to analyze changes in a conversationas the conversation progresses, and then going back and weightingearlier portions of the conversation states with a correction factordetermined by later states.

Even though prior research has presented instances of highlightingcertain entered data in a visual form by calculating the data'simportance to the context, a need still exists in the art of neuralnetworks for automated virtual assistants to address an issue that thestandard HAN procedure omits—creating more information in real timevisual displays of text, color and graphics when the HAN procedures ofthe prior art yield uniform weights to all portions of a data set.

SUMMARY

A computerized method for highlighting relative importance of portionsof a conversation displayed on a graphical user interface includesstoring the conversation in computerized memory connected to a computerprocessor that is configured to display conversations on a graphicaluser interface, wherein a display of the conversation illustratesconversation data according to respective conversation participants'turns in providing conversation input. The method further includesweighting respective turns of the conversation by providing theconversation input of the respective turns to a hierarchical attentionnetwork stored in the memory, wherein the hierarchical attention networkuses the processor to calculate sequential long-short-term-memory cells(LSTM) in the memory. By using later LSTM cell data to update weightingvalues for prior LSTM cell data in a sequence of turns of conversationinput data, the method includes displaying the conversationparticipants' turns on the graphical user interface with a visibleindication of respective weighting values for each of the participants'turns.

A computerized method according to this disclosure includes non-limitingembodiments wherein weighting the respective turns comprises addingconversation input data from additional later turns of the conversationto new LSTM cells;

A computerized method according to this disclosure includes non-limitingembodiments wherein weighting the respective turns comprises changingweights of the prior LSTM cell data in response to the additional laterturns;

A computerized method according to this disclosure includes non-limitingembodiments wherein the processor identifies a plurality of turns in theconversation illustrating at least one change in weight distributionamong the plurality of turns as an attention dependency switch;

A computerized method according to this disclosure includes non-limitingembodiments wherein the processor identifies sequential turns in theconversation illustrating at least one change in weight between twoturns as a context dependency switch;

A computerized method according to this disclosure includes non-limitingembodiments wherein the processor identifies at least one turn in theconversation illustrating at least one change in weight, across theentire conversation and greater than a variation dependency variable, asa variation dependency switch;

A computerized method according to this disclosure includes non-limitingembodiments wherein weighting a group of turns in the conversationcomprises forming a weight vector from occurrences of at least oneattention dependency switch, at least one context dependency switch, andat least one variation dependency switch, averaging components of thevector, and representing each term in the group of terms on thegraphical user interface with a pixel intensity that corresponds to theaverage of the components of the weight vector;

A computerized method according to this disclosure includes non-limitingembodiments wherein the hierarchical attention network uses theprocessor to calculate sequential long-short-term-memory cells (LSTM) inthe memory when a prior weighting of turns in a conversation have had adegree of uniformity greater than a uniformity tolerance threshold;

A computerized method according to this disclosure includes non-limitingembodiments wherein the processor and memory form a turn weight vectorcomprising weighting values for turns in the conversation and calculatea degree of uniformity (a) across members of the vector;

A computerized method according to this disclosure includes non-limitingembodiments wherein the processor and the memory use the turn weightvector to identify either uniformity or non-uniformity across theweights in the weight vector by comparing sequential weighting vectorsfrom sequential turns to an attention dependency variable (τa);

A computerized method according to this disclosure includes non-limitingembodiments wherein the processor and the memory use the turn weightvector to identify instances across the conversation in which anaddition of a turn changes the weights of previous turns by comparingthe weighting vectors to a context dependency variable (τc);

A computerized method according to this disclosure includes non-limitingembodiments wherein the processor and the memory use the turn weightvector to identify individual weighting value changes across theconversation in which an addition of a turn changes the weight of arespective individual weighting value more than variation dependencyvariable (τv);

A computerized method according to this disclosure includes non-limitingembodiments wherein displaying the conversation participants' turns onthe graphical user interface with a visible indication of respectiveweighting values comprises:

the processor and memory forming a turn weight vector comprisingweighting values for turns in the conversation and calculating a degreeof uniformity (α) across members of the vector;

the processor and the memory using the turn weight vector to identifyattention dependency and either uniformity or non-uniformity across theweighting values in the weight vector by comparing sequential weightingvectors from sequential turns to an attention dependency variable (τa).

the processor and the memory using the turn weight vector to identifyinstances across the conversation in which an addition of a turnidentifies context dependency and changes in the weighting values ofprevious turns by comparing the weighting vectors to a contextdependency variable (τc);

the processor and the memory use the turn weight vector to identifyvariation dependency and individual weighting value changes across theconversation, in which an addition of a turn changes the weight of arespective individual weighting value more than variation dependencyvariable (τv); and

selecting the visible indication for displaying the respective turnsaccording to combinations of attention dependency, context dependency,and variation dependency across the turn weight vector for theconversation;

A computerized method according to this disclosure includes non-limitingembodiments wherein a selection of the visible indication of respectiveweighting values for each of the participants' turns is updated in realtime during the conversation;

A computerized method according to this disclosure includes non-limitingembodiments wherein a point of escalation in the conversation isidentified from the weighting values; and

A computerized method according to this disclosure includes non-limitingembodiments wherein the conversation comprises at least one of textdata, voice data, natural language data derived from voice data, andgraphics data.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication with thecolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. Reference will now be made to the accompanyingdrawings, which are not necessarily drawn to scale.

FIG. 1 is a schematic representation of a graph showing a comparisonbetween prior art hierarchical attention network data and hierarchicalnetwork data of this disclosure.

FIG. 2 is a PRIOR ART schematic representation of a hierarchicalattention network as illustrated in references noted at the end of thisdisclosure to Yang et al. 2016.

FIG. 3 is a schematic representation of conversation data separated into turns from each conversation participant and adjusted as additionalturns are added to the memory.

FIG. 4 is a schematic representation of conversation data separated into turns from each conversation participant and adjusted as additionalturns are added to the memory.

FIG. 5 is a schematic representation of conversation data separated into turns from each conversation participant and adjusted as additionalturns are added to the memory.

FIG. 6 is a schematic representation of conversation data separated into turns from each conversation participant and adjusted as additionalturns are added to the memory.

FIG. 7 is a representation of a computer environment in which the methodof this disclosure may be practiced.

FIG. 8 is a tabular schematic view of a series of conversation turnsbetween a caller and an interactive virtual assistant and uses black andwhite colored boxes to illustrate how weights of each turn can beuniform or non-uniform depending on the number and content of turns inthe conversation.

FIG. 9 is a tabular schematic view of a series of conversation turnsbetween a caller and an interactive virtual assistant and uses black andwhite colored boxes to illustrate how weights of each turn arenon-uniform when using embodiments of this disclosure.

DETAILED DESCRIPTION

The following description of the disclosure is provided as an enablingteaching of the disclosure in its best, currently known embodiment(s).To this end, those skilled in the relevant art will recognize andappreciate that many changes can be made to the various embodiments ofthe embodiments described herein, while still obtaining the beneficialresults of the present disclosure. It will also be apparent that some ofthe desired benefits of the present disclosure can be obtained byselecting some of the features of the present disclosure withoututilizing other features. Accordingly, those who work in the art willrecognize that many modifications and adaptations to the presentdisclosure are possible and can even be desirable in certaincircumstances and are a part of the present disclosure. Thus, thefollowing description is provided as illustrative of the principles ofthe present disclosure and not in limitation thereof.

Terminology

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood to one of ordinary skill inthe art to which this disclosure belongs.

As used in the specification and claims, the singular form “a,” “an,”and “the” include plural references unless the context clearly dictatesotherwise. For example, the term “an agent” includes a plurality ofagents, including mixtures thereof.

As used herein, the terms “can,” “may,” “optionally,” “can optionally,”and “may optionally” are used interchangeably and are meant to includecases in which the condition occurs as well as cases in which thecondition does not occur. Thus, for example, the statement that aformulation “may include an excipient” is meant to include cases inwhich the formulation includes an excipient as well as cases in whichthe formulation does not include an excipient.

Ranges can be expressed herein as from “about” one particular valueand/or to “about” another particular value. When such a range isexpressed, another embodiment includes from the one particular valueand/or to the other particular value. Similarly, when values areexpressed as approximations, by use of the antecedent “about,” it willbe understood that the particular value forms another embodiment. Itwill be further understood that the endpoints of each of the ranges aresignificant both in relation to the other endpoint, and independently ofthe other endpoint. It is also understood that there are a number ofvalues disclosed herein, and that each value is also herein disclosed as“about” that particular value in addition to the value itself. Forexample, if the value “10” is disclosed, then “about 10” is alsodisclosed.

Publications cited herein are hereby specifically incorporated byreference in their entireties and at least for the material for whichthey are cited.

In the discussions below, a “turn” refers to back and forth portions ofa dialog in which one of the users of a system is providing content,whether online, typed, voice, data, graphics, or any othercommunications.

The discussion herein discloses uses for conversation “c” that wouldinclude all kinds of conversation data, without limitation, being textdata, voice data, graphical data or information that has been created bynatural language processing functions.

The term conversation is used in its broadest sense to include thepassing of information between two participants, where either or both ofthe participants may be computerized or human agents. The participantsin a conversation take “turns” providing conversation input, and the useof the term “turns” is not limited in any respect related to time, asthe turns can be sequential or contemporaneous to any degree or atexactly the same time.

The references to LSTM cell data by the words “later” and “prior” isonly for example purposes to note that the cell data is stored in adifferent portion of memory (i.e., the later and prior data aredistinct). The later and prior cell data can be viewed in a temporalcontext with prior data being entered into a network before later celldata, but that is not limiting of the disclosure. This is particularlytrue when methods disclosed herein utilize forward and backwardprocesses along any given string of data at issue.

In one scenario, a user and an automated computer system each take turnscommunicating in an online or electronic communication session. Tomaintain quality of service, users are transferred to humanrepresentatives when their conversations with an intelligent virtualassistant (IVA) fail to progress. These transfers are known as“escalations.” This application involves, in one non-limitingembodiment, the escalation of conversations or data entry such as, butnot limited to, those recorded as Internet chats, online help sessions,natural language processing of voice data, and the like. This disclosureapplies HAN to such conversations in a sequential manner by feeding eachuser turn in a customer interaction to HAN as they occur, to determineif the conversation should escalate. If so, the user will be transferredto a live chat representative to continue the conversation. To help thehuman representative quickly determine the cause of the escalation, thedescribed embodiments generate a visualization of the user's turns usingthe attention weights to highlight the turns influential in theescalation decision. This helps the representative quickly scan theconversation history and determine the best course of action based onproblematic turns. Unfortunately, there are instances where theattention weights for every turn at the point of escalation are nearlyequal, requiring the representative to carefully read the history todetermine the cause of escalation unassisted. Table 1 shows one suchexample with uniform attention weights at the point of escalation.

This disclosure notes that the visualizations are generated in real-timeat the point of escalation. The user must wait for the humanrepresentative to review the IVA chat history and resume the failedtask. Therefore, the described embodiments seek visualization methodsthat do not add significant latency to the escalation transfer. Usingthe attention weights for turn influence is fast as they were alreadycomputed at the time of classification. However, these weights will notgenerate useful visualizations for the representatives when their valuesare similar across all turns (see HAN Weight in Table 1). To overcomethis problem, the described embodiments develop a method that producesmore informative visuals (see “OUR” Disclosure Weight in Table 1) usingthe changes in turn importance over cumulative turns, instead of rawturn weights.

TABLE 1 TURN USER TEXT HAN WEIGHT OUR WEIGHT 1 Is there a customer 0.30.3 service phone number 2 I just requested a 0.3 0.3 refund throughExpedia because I picked the wrong flight day, how long will it take 3Is there a way to 0.3 0.6 expedite that 4 Can we rush the 0.3 0.6 refundbecause I need to book another ticket 5 Refunds 0.3 1.0

In Table 1, above, the influence on escalation of each user turn in aconversation. Higher weight turns are displayed as being darker incolor. As the HAN weights are uniform, and, therefore, similar in color,it is difficult to infer the cause of escalation. In contrast, accordingto this disclosure, the weights of the visual noted as “OUR Weight” onthe same conversation show distinct turn importance, thus, quicklyindicating the cause of escalation in this conversation.

To determine when the turn weights are uniform, the describedembodiments use perplexity (Brown et al., 1992), the exponential of theentropy. Since entropy is a measure of the degree of randomness,perplexity is then a measure of the number of choices that comprise thisrandomness. If a conversation c escalates on turn i with attentionweights [w1;w2; . . . w_(i)], let αc=i−perplex(w1;w2; . . . wi).Intuitively, a should be low when uniformity is high.

This disclosure measures the a of every escalated conversation anddefines a uniformity threshold for a (FIG. 1). For example, if the athreshold for uniformity is 0.5, 20% of conversations in the datasetwill result in HAN visuals where all turns have similar weight; thus, nomeaningful visualization can be produced. Companies that deploy IVAsolutions for customer service report escalated conversation volumes of1,100 per day for one customer (Next IT, 2018). Therefore, even at 20%,contact centers handling multiple companies may see hundreds orthousands of conversations per day with no visualizations. However, ifthe described embodiments use the embodiments described herein ininstances where HAN weights are uniform, the conversations are allnon-uniform using the same α=0.5.

In FIG. 8, the described embodiments see the bottom visualization wherethe weights are uniform at the point of escalation. However, on the 2ndturn, the H_(AN) had produced more distinct weights. At the point ofescalation on turn 3 in FIG. 8, the weights become uniform. It is clearfrom this example that the importance of a single turn can changedrastically as the conversation progresses. Using these changes in turnattention, the described embodiments formalize a set of rules to createan alternative visualization for the entire conversation in cases wherethe attention weights are uniform.

For every conversation of length n turns, when turn n+1 is added, thedescribed embodiments consider three forms of behavior that help tocreate a new visual: attention, context, and variation dependencyswitches. An attention dependency switch occurs when the addition of aturn changes the distribution of weights. Suppose the describedembodiments have a 4-turn conversation. In FIG. 5, considering only thefirst 3 turns gives a uniform distribution of weights (left). However,when the described embodiments add turn 4 (FIG. 5, right), thedistribution shifts to one of non-uniformity. The described embodimentsconsider the addition of any such turn that causes a switch from uniformto non-uniform or vice-versa in the creation of visuals.

A context dependency switch occurs when the addition of a turn causes aprevious turn's weight to change significantly. In FIG. 6, the additionof turn 6 causes turn 3's weight to spike.

The final switch of consideration is a variation dependency switch,which occurs when the weight of some turn i changes significantly overthe entire course of a conversation. Note that this differs from contextdependency because the latter determines turn i's change with theaddition of only one turn. For determining attention dependency, thedescribed embodiments consider normalized attention weights, but forvariation and context, the described embodiments consider theunnormalized output logits from the HAN. It is also important to notethat an attention dependency switch can occur without a contextdependency switch and vice-versa.

Escalation data described herein was obtained from (Freeman and Beaver,2017), which consists of 7754 conversations (20; 808 user turns) fromtwo commercial airline IVAs. 1268 of the 7754 conversations had beentagged for escalation. The classifier (HAN) used for escalationprediction is outlined in (Yang et al., 2016). Given the occurrences ofattention (μ), context (β), and variation (γ) dependency switches, thedescribed embodiments show a procedure on how a visual of the entireconversation can be created. For each turn Ti, the described embodimentscreate a vector vi=[μ_(i); β_(i); γ_(i)], where each variable insidethis vector takes the value 1 when the attention, context, and variationdependency switches trigger, respectively, and 0 otherwise. Thedescribed embodiments compute mean vi=(μ_(i)+β_(i)+γ_(i))/3, and usethis value to represent the intensity of a single color (blue in Table1). The higher the value of mean vi, the higher the color intensity.

After removing one turn conversations (as they are uniform by default),the number of turns that had a context dependency switch as a result ofadding a new turn was 4,563. However, the number of times that such anevent coincided at least once with escalation was 766. As it appearedthat the effect of context dependency was quite low, the describedembodiments next considered the variation and attention dependencyvariables. The total number of turns that had a variation dependencyswitch was 2,536, and 1,098 also coincided with a change of escalation,indicating that a variation dependency switch is potentially valuable inthe creation of new visuals. In addition, the number of uniform tonon-uniform turn pairs (uniform weight distribution for first i turnsbut non-uniform for first i+1 turns) was 1; 589 whereas the number ofnon-uniform to uniform turn pairs was 259. Out of the times when therewas a uniform to non-uniform switch, 710 cases coincided with escalationcompared to only 22 for non-uniform to uniform changes.

As shown in FIG. 1, the addition of the disclosed method when the H_(AN)weights are uniform greatly reduces or even eliminates the uniformity atlower α thresholds. To determine if the visuals were also assigningweights properly, the described embodiments had three reviewers rate ona 0 to 10 scale (0 being poor, 10 being best) of how well eachvisualization highlights the influential turns for escalation in theconversation (annotator instructions available in supplement). See FIG.9 for an example that was tagged nearly perfectly by reviewers.

From the 1,268 conversations that escalated in the dataset, thedescribed embodiments first filtered conversations by a uniformitythreshold, a. The described embodiments chose an a value of 0:18empirically as follows. Three reviewers tagged 100 random escalatedconversations for uniformity of weights at various a thresholds. Asexpected, the weights appeared less uniform as a increased. For example,at α=0.5, weights such as [0.2; 0.2; 0.6] would be considered uniformwhich all three reviewers disagreed with. However, the reviewers allagreed that weight distributions below the 0:18 threshold appeareduniform 90% of the time, which the described embodiments considered goodagreement. At this threshold, 10.9% or 138 conversations remained.

Next, the described embodiments filtered the conversations that were notcorrectly classified by H_(AN), leaving 85 or 6.7%. The average 0-10rating between the three reviewers over the remaining conversations was6. This demonstrates that on average, reviewers felt that thevisualizations were adequate. Put in perspective, adding adequatevisuals to the thousands of daily escalations that would otherwise haveno visual is a great improvement. This may reduce the summarization timefor human representatives, and, therefore, user wait time on transfer.

In addition to the possible reduction in human review time, the visualsonly required 0.9 milliseconds on average to compute per conversation.This adds insignificant latency to the transfer while generating thevisualization, which is an important goal.

Perplexity

For a probability distribution D over the sample space, the perplexitymeasure is defined as the exponential of the entropy of D. Moreformally, perplex(D)=2^(H(D))), where the entropy is

${H(D)} = {\Sigma_{x \in \Omega}{D(x)}\mspace{14mu}\log_{2}{\frac{1}{D(x)}.}}$

As entropy is a measure of the degree of randomness in D, perplexity isa measure of the number of choices that comprise this randomness. Thefollowing properties of perplexity will be applicable.

1. For any distribution D, the value of perplex(D) is always positive.(2^(X)>0 for all x.)

2. For any distribution D over N values, the described embodiments haveperplex(D)≤N. The larger the value, the closer D is to being uniform.The equality holds if and only if D is uniform.

With respect to property (2) above, the described embodiments define ametric α_(N)(D)=N−perplex(D), where D is any distribution over N values.Thus, for all N≥1 and all distributions D that are uniform over Nvalues, it must be the case that αN(D)=0. Furthermore, α_(N) (D)≥0 forall N and D. The described embodiments drop the subscript N fromα_(N)(D) when it is obvious from the context.

Perplexity Based Measure of Uniformity

In this disclosure, obtaining an exact uniform distribution isinefficient. In such cases, it suffices to consider a distribution to beuniform if it is almost the same over all values. In other words, agiven distribution D on N values is τ-uniform if α_(N)(D)≤τ. Note thatsince α_(N)(D) can be at most N−1 (as N≥1), this restricts τ to be anyreal number between 0 and N−1. In this context, given a distribution Dover N values, the described embodiments will refer to α(D) as themeasure of uniformity of D. The smaller the value of a (D), the closer Dis to being uniform.

Given a conversation C that contains N turns, let w_(i) be the vector ofattention weights obtained from inputting T₁, . . . T_(i) (where T_(i)is the i-th turn in C) to H_(AN).

Attention Dependency Variable Change

There is an attention dependency variable change from turn T_(i) toT_(i+1) with some threshold Ta if any one of the following occurs:

α(w _(i+1))≥τ_(a) and α(w _(i))<τ_(a)  1.

α(w _(i))≥τ_(a) and α(w _(i+1))<τ_(a)  2.

With 1, the described embodiments are switching from a uniformdistribution to a non-uniform distribution with the addition of turnT_(i+1). With 2, the described embodiments are switching from anon-uniform distribution to a uniform distribution.

This τa is chosen in such a way to determine approximate uniformity asmentioned in section 2 as it is inefficient for this disclosure.

Note that it is possible that the attention dependency variable changeis observed for many turns and not just one.

Context Dependency Variable Change

To capture the event in which the addition of a turn changes the weightsof some previous turn significantly, the described embodiments use acontext dependency variable. There is a context dependency variablechange in turn T_(j) by addition of turn T_(i) for j<i with thresholdτ_(c)>0 if

|w _(i+1)[j]−w _(i)[j]|≥τ_(c).

Note that wi[j] denotes the weight of turn Tj in wi.

Variation Dependency Variable Change

Finally, the described embodiments capture how the weight of a giventurn changes every time a new turn is appended to the conversation. Thisdiffers from context dependency, where the described embodimentsconsider the effect of adding only one turn. There is a variabledependency variable change in turn T_(i) with some threshold_v>0 whenthe conversation has N turns if

${\frac{1}{N - i}{\sum\limits_{k = i}^{N - 1}\;{{{w_{k}\lbrack i\rbrack} - {w_{k + 1}\lbrack i\rbrack}}}}} \geq \tau_{v}$

In testing the operations of this disclosure, the described embodimentscompute the thresholds mentioned in the definitions above as follows:

1. For attention dependency, the described embodiments experimented withvarious τ_(a) thresholds and tagged 100 randomly chosen conversationsfor each of those thresholds to determine potential candidates. Forexample, using a threshold of 0.5, weight vectors such as [0.2; 0.2;0.6] would be considered uniform, which the described embodimentsgreatly disagreed with. However, the reviewers all agreed that weightdistributions below the 0:18 threshold appeared uniform 90% of the time,which the described embodiments considered good agreement.

2. For context dependency and variation dependency switches, thedescribed embodiments chose the value of τ_(c) and τ_(v), respectively,using the 75th percentile of the values for different turns. Uponcomparison with manual tagging of 100 randomly chosen conversations, theannotators were observed to agree on all 100 cases for the contextdependency switch and 99 out of 100 cases for the variation dependencyswitch.

In FIG. 3, an attention dependency switch has occurred (uniform tonon-uniform distribution), but there is no context dependency variablechange. In FIG. 4, a context dependency variable change has occurred asmany previous weights have spiked, but the distribution of weights hasnot changed (no attention dependency variable change).

In one non-limiting embodiment, a hierarchical attention network wasimplemented with TensorFlow (Abadi et al., 2015) and has substantiallythe same architecture as in (Yang et al., 2016) with the exception thatLSTM cells are used in place of GRU. The described embodiments used the200-dimensional word embeddings from glove.twitter.27B (Pennington etal., 2014) and did not adapt them during the training of our model.

Each recurrent encoding layer had 50 forward and 50 backward cells,giving 100-dimensional embeddings each for turns and conversations. Inpredicting escalation, the network obtained an Fi of 81.31+/−0:94%(87:71+/−3:17% precision, 75.90+/−2:61% recall, averaged over fiverandom splits). To compute these metrics, turn-level annotations wereconverted to conversation-level annotations by labeling a conversationescalate if any turn in the conversation was labeled escalate.

For the visualization experiments, a random 80-20 split was used tocreate training and testing sets. The training set consisted of 6,203conversations of which 1,027 should escalate. The testing set consistedof 1; 551 conversations of which 241 should escalate.

Annotator Instructions

The annotators were instructed to rate on a scale of 0 to 10 of howwell, in their opinion, each visualization highlights the influentialuser turns for escalation in a given conversation. They were remindedthat if a turn is given a darker color, then that turn supposedly hasgreater influence in determining escalation. They were, thus, given thetask of determining if they agree with the visualization's decision. Arating of 0 was instructed to be given on complete disagreement, and 10upon perfect agreement. Previously, the described embodiments considereda technique which used a single color with varying intensity tohighlight when the different switches are triggered. Recall that foreach turn T_(i), the described embodiments create a vector v_(i)=[μ_(i),β_(i), γ_(i)], where each variable inside this vector takes the value 1when the attention, context, and variation dependency switches trigger,respectively, and 0 otherwise. An alternate approach is to use thevector v_(i) to create an RGB visual (see Table 2), in which thedescribed embodiments obtain as follows. Table 2 is an example of a RGBvisual. The turn color denotes what combination of attention weightbehaviors were triggered in the turn. The described embodiments computethe RGB value of the color used to denote T_(i) by setting r=255

when μ_(i)=1,b=255 if β_(i)=1 and g=255 if

γ_(i)=1.

TABLE 2 TURN USER TEXT RGB Weight 1 How do I change White my ticket? 2bwrlzd Green 3 How do I change Black my ticket? 4 I would like to Bluechange my destination airport 5 Same day flight Yellow changes

If any of v_(i)'s elements are 0, its respective color value is set to0. Based on these values, the final color denotes what combination ofthese three values in the vector are represented. Unlike the monochromevisual, this visualization scheme allows the user to look at the colorsand deduce the events that T_(i) must have triggered. However, thisvisual is more complex for a reader to understand and requires constantaccess to a color legend (see Table 3) for interpretation.

TABLE 3 attention context variation color X Red X Green X Dk Blue X XYellow X X Pink X X Lt Blue White X X X Black

Since one non-limiting goal here is to produce user-friendly visualswhich can be interpreted quickly, the described embodiments only suggestthe RGB visual as an alternative and use the monochrome visual in mostapplications.

In accordance with the above noted embodiments, a computerized methodfor highlighting relative importance of portions of a conversationdisplayed on a graphical user interface includes storing theconversation “c” in computerized memory (204) connected to a computerprocessor (602) that is configured to display conversations on agraphical user interface (217), wherein a display of the conversationillustrates conversation data according to respective conversationparticipants' turns (1, 2, 3, 4, 5, 6, et seq.) in providingconversation input. The method further includes weighting respectiveturns of the conversation by providing the conversation input of therespective turns to a hierarchical attention network stored in thememory, wherein the hierarchical attention network uses the processor tocalculate sequential long-short-term-memory cells (LSTM) (308, 408, 508,608) in the memory. By using later LSTM cell data (e.g., 316, 417, 517,618) to update weighting values for prior LSTM cell data (e.g., 315,416, 516, 617) in a sequence of turns of conversation input data, themethod includes displaying the conversation participants' turns on thegraphical user interface (217) with a visible indication of respectiveweighting values for each of the participants' turns.

The methods and computer implemented algorithms for processingconversation data according to this disclosure may be implemented as asystem via the above described computer equipment, such as a processor(202), memory (204), and graphical user interface (217). The memory mayhave software stored thereon, or accessible via a network to implementthe computerized methods and computer implemented algorithms of thisdisclosure. Any single memory device may be considered a computerprogram product carrying the software and instructions to implement thedescribed methods.

With reference to FIG. 7, an exemplary system for implementing aspectsdescribed herein includes a computing device, such as computing device200. In its most basic configuration, computing device 200 typicallyincludes at least one processing unit 202 and memory 204. Depending onthe exact configuration and type of computing device, memory 204 may bevolatile (such as random access memory (RAM)), non-volatile (such asread-only memory (ROM), flash memory, etc.), or some combination of thetwo. This most basic configuration is illustrated in FIG. 7 by dashedline 206.

Computing device 200 may have additional features/functionality. Forexample, computing device 200 may include additional storage (removableand/or non-removable) including, but not limited to, magnetic or opticaldisks or tape. Such additional storage is illustrated in FIG. 7 byremovable storage 208 and non-removable storage 210.

Computing device 200 typically includes a variety of computer readablemedia. Computer readable media can be any available media that can beaccessed by the device 200 and includes both volatile and non-volatilemedia, removable and non-removable media.

Computer storage media include volatile and non-volatile, and removableand non-removable media implemented in any method or technology forstorage of information such as computer readable instructions, datastructures, program modules or other data. Memory 204, removable storage208, and non-removable storage 610 are all examples of computer storagemedia. Computer storage media include, but are not limited to, RAM, ROM,electrically erasable program read-only memory (EEPROM), flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bycomputing device 200. Any such computer storage media may be part ofcomputing device 200.

Computing device 200 may contain communication connection(s) 612 thatallow the device to communicate with other devices. Computing device 200may also have input device(s) 614 such as a keyboard, mouse, pen, voiceinput device, touch input device, etc. Output device(s) 616 such as adisplay (217), speakers, printer, etc. may also be included. All thesedevices are well known in the art and need not be discussed at lengthhere.

It should be understood that the various techniques described herein maybe implemented in connection with hardware components or softwarecomponents or, where appropriate, with a combination of both.Illustrative types of hardware components that can be used includeField-programmable Gate Arrays (FPGAs), Application-specific IntegratedCircuits (ASIC s), Application-specific Standard Products (ASSPs),System-on-a-chip systems (SOCs), Complex Programmable Logic Devices(CPLDs), etc. The methods and apparatus of the presently disclosedsubject matter, or certain aspects or portions thereof, may take theform of program code (i.e., instructions) embodied in tangible media,such as floppy diskettes, CD-ROMs, hard drives, or any othermachine-readable storage medium where, when the program code is loadedinto and executed by a machine, such as a computer, the machine becomesan apparatus for practicing the presently disclosed subject matter.

Although exemplary implementations may refer to utilizing aspects of thepresently disclosed subject matter in the context of one or morestand-alone computer systems, the subject matter is not so limited, butrather may be implemented in connection with any computing environment,such as a network or distributed computing environment. Still further,aspects of the presently disclosed subject matter may be implemented inor across a plurality of processing chips or devices, and storage maysimilarly be effected across a plurality of devices. Such devices mightinclude personal computers, network servers, and handheld devices, forexample.

REFERENCES

-   Leila Arras, Franziska Horn, Gregoire Montavon, Klaus-Robert Muller,    and Wojciech Samek. 2017a. “what is relevant in a text document?”:    An interpretable machine learning approach. PloS one 12(8):e0181142.-   Leila Arras, Gregoire Montavon, Klaus-Robert Muller, and Wojciech    Samek. 2017b. Explaining recurrent neural network predictions in    sentiment analysis. In Proceedings of the 8th Workshop on    Computational Approaches to Subjectivity, Sentiment and Social Media    Analysis. ACL, pages 159-168.-   Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick    Klauschen, Klaus-Robert Muller, and Wojciech Samek. 2015. On    pixel-wise explanations for non-linear classifier decisions by    layer-wise relevance propagation. PloS one 10(7):e0130140.-   Dzmitry Bandanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural    machine translation by jointly learning to align and translate.    arXiv preprint arXiv:1409.0473.-   Peter F Brown, Vincent J Della Pietra, Robert L Mercer, Stephen A    Della Pietra, and Jennifer C Lai. 1992. An estimate of an upper    bound for the entropy of english. Computational Linguistics    18(1):31-40.-   Cynthia Freeman and Ian Beaver. 2017. Online proactive escalation in    multi-modal automated assistants. In FLAIRS 2017. AAAI Press.-   Gregoire Montavon, Wojciech Samek, and Klaus Robert Muller. 2017.    Methods for interpreting and understanding deep neural networks.    Digital Signal Processing. Next-IT. 2018.

Next it case studies. http://nexit.com/case-studies.

-   Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why    should i trust you?: Explaining the predictions of any classifier.    In Proceedings of the 22nd ACM SIGKDD International Conference on    Knowledge Discovery and Data Mining. ACM, pages 1135-1144.-   Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul    Kundaje. 2016. Not just a black box: Learning important features    through propagating activation differences. arXiv preprint    arXiv:1605.01713.-   Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J Smola,    and Eduard H Hovy. 2016. Hierarchical attention networks for    document classification. In HLT-NAACL. pages 1480-1489.-   Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng    Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean,    Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp,    Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz,    Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane,    Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster,    Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul    Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol    Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and    Xiaoqiang Zheng. 2015. Tensor-Flow: Large-scale machine learning on    heterogeneous systems. Software available from tensorflow.org.    https://www.tensorflow.org/. Jeffrey Pennington, Richard Socher, and    Christopher D. Manning. 2014. Glove: Global vectors for word    representation. In Empirical Methods in Natural Language Processing    (EMNLP). pages 1532-1543. http://www.aclweb.org/anthology/D14-1162.-   Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J Smola,    and Eduard H Hovy. 2016. Hierarchical attention networks for    document classification.-   In HLT-NAACL. pages 1480-1489.

It should be understood that while the present disclosure has beenprovided in detail with respect to certain illustrative and specificaspects thereof, it should not be considered limited to such, asnumerous modifications are possible without departing from the broadspirit and scope of the present disclosure as defined in the appendedclaims. It is, therefore, intended that the appended claims cover allsuch equivalent variations as fall within the true spirit and scope ofthe embodiments claimed herein.

1. A computerized method for highlighting relative importance ofportions of a conversation displayed on a graphical user interface,comprising: storing the conversation in text form in computerized memoryconnected to a computer processor that is configured to displayconversation text on a graphical user interface, wherein a display ofthe conversation illustrates conversation data according to respectiveconversation participants' turns in providing conversation input;weighting respective turns of the conversation by providing theconversation input of the respective turns to a hierarchical attentionnetwork stored in the memory, wherein the hierarchical attention networkuses the processor to calculate sequential long-short-term-memory cells(LSTM) in the memory; using later LSTM cell data to update weightingvalues for prior LSTM cell data in a sequence of turns of conversationinput data; and displaying the conversation participants' turns on thegraphical user interface with a visible indication of respectiveweighting values for each of the participants' turns.
 2. A computerizedmethod according to claim 1, wherein weighting the respective turnscomprises adding conversation input data from additional later turns ofthe conversation to new LSTM cells.
 3. A computerized method accordingto claim 2, wherein weighting the respective turns comprises changingweights of the prior LSTM cell data in response to the additional laterturns.
 4. A computerized method according to claim 3, wherein theprocessor identifies a plurality of turns in the conversationillustrating at least one change in weight distribution among theplurality of turns as an attention dependency switch.
 5. A computerizedmethod according to claim 3, wherein the processor identifies sequentialturns in the conversation illustrating at least one change in weightbetween two turns as a context dependency switch.
 6. A computerizedmethod according to claim 3, wherein the processor identifies at leastone turn in the conversation illustrating at least one change in weight,across the entire conversation and greater than a variation dependencyvariable, as a variation dependency switch.
 7. A computerized methodaccording to claim 2, wherein weighting a group of turns in theconversation comprises forming a weight vector from occurrences of atleast one attention dependency switch, at least one context dependencyswitch, and at least one variation dependency switch, averagingcomponents of the vector, and representing each term in the group ofterms on the graphical user interface with a pixel intensity thatcorresponds to the average of the components of the weight vector.
 8. Acomputerized method according to claim 2, wherein the hierarchicalattention network uses the processor to calculate sequentiallong-short-term-memory cells (LSTM) in the memory when a prior weightingof turns in a conversation have had a degree of uniformity greater thana uniformity tolerance threshold.
 9. A computerized method according toclaim 8, wherein the processor and memory form a turn weight vectorcomprising weighting values for turns in the conversation and calculatea degree of uniformity (a) across members of the vector.
 10. Acomputerized method according to claim 9, wherein the processor and thememory use the turn weight vector to identify either uniformity ornon-uniformity across the weights in the weight vector by comparingsequential weighting vectors from sequential turns to an attentiondependency variable (Ta).
 11. A computerized method according to claim9, wherein the processor and the memory use the turn weight vector toidentify instances across the conversation in which an addition of aturn changes the weights of previous turns by comparing the weightingvectors to a context dependency variable (τc).
 12. A computerized methodaccording to claim 9, wherein the processor and the memory use the turnweight vector to identify individual weighting value changes across theconversation in which an addition of a turn changes the weight of arespective individual weighting value more than variation dependencyvariable (τv).
 13. A computerized method according to claim 1, whereindisplaying the conversation participants' turns on the graphical userinterface with a visible indication of respective weighting valuescomprises: the processor and memory forming a turn weight vectorcomprising weighting values for turns in the conversation andcalculating a degree of uniformity (α) across members of the vector; theprocessor and the memory using the turn weight vector to identifyattention dependency and either uniformity or non-uniformity across theweighting values in the weight vector by comparing sequential weightingvectors from sequential turns to an attention dependency variable (τa).the processor and the memory using the turn weight vector to identifyinstances across the conversation in which an addition of a turnidentifies context dependency and changes in the weighting values ofprevious turns by comparing the weighting vectors to a contextdependency variable (τc); the processor and the memory use the turnweight vector to identify variation dependency and individual weightingvalue changes across the conversation, in which an addition of a turnchanges the weight of a respective individual weighting value more thanvariation dependency variable (τv); and selecting the visible indicationfor displaying the respective turns according to combinations ofattention dependency, context dependency, and variation dependencyacross the turn weight vector for the conversation.
 14. A computerizedmethod according to claim 1, wherein a selection of the visibleindication of respective weighting values for each of the participants'turns is updated in real time during the conversation.
 15. Acomputerized method according to claim 1, wherein a point of escalationin the conversation is identified from the weighting values.
 16. Acomputerized system comprising the method of claim 1 implemented insystem hardware comprising the processor, memory, and graphical userinterface.